Skip to content

OPC UA Part 4 6.6 Redundancy (server + client) with opt-in distributed high-availability#3918

Draft
marcschier wants to merge 67 commits into
OPCFoundation:masterfrom
marcschier:nodestatestorage
Draft

OPC UA Part 4 6.6 Redundancy (server + client) with opt-in distributed high-availability#3918
marcschier wants to merge 67 commits into
OPCFoundation:masterfrom
marcschier:nodestatestorage

Conversation

@marcschier

@marcschier marcschier commented Jun 26, 2026

Copy link
Copy Markdown
Collaborator

Summary

This branch implements OPC 10000-4 §6.6 Redundancy end to end — server and client, transparent and non-transparent, across all failover modes (None, Cold, Warm, Hot, HotAndMirrored, Transparent) — and layers an opt-in distributed high-availability (HA) capability on top so replicas can share address-space, session, and subscription state and expose redundancy to clients through the documented OPC UA mechanisms (for example, running a server replica set across nodes in Kubernetes).

The single-instance, in-memory path stays zero-overhead: every distributed feature is opt-in through dependency injection, with a direct-construction fallback, and replaced APIs are kept and marked [Obsolete].

New packages

Package Purpose
OPCFoundation.NetStandard.Opc.Ua.Server.Redundancy §6.6 redundancy nodes plus opt-in distributed state: shared address space, secure session sharing, subscription/continuation-point mirroring, and leader election.
OPCFoundation.NetStandard.Opc.Ua.Server.Redundancy.Crdt Active/active, leaderless multi-writer replication via CRDTs (an extension beyond §6.6).
OPCFoundation.NetStandard.Opc.Ua.Server.Redundancy.K8s Kubernetes integration: Lease leader election, EndpointSlice peer discovery, and ServiceLevel-driven readiness.

Also adds the worked samples Applications/RedundantServer and Applications/RedundantClient, and the guides Docs/HighAvailability.md and Docs/HighAvailabilityKubernetes.md.

Server side (§6.6)

  • AddServerRedundancy(...) populates the live Server.ServerRedundancy nodes (RedundancySupport plus RedundantServerArray / ServerUriArray / CurrentServerId) and drives Server.ServiceLevel from an IServiceLevelProvider (leader high, standby low). Server.ServerRedundancy is typed to the NonTransparent or Transparent subtype according to the configured mode.
  • AddRequestServerStateChange(...) implements the §6.6.5 manual-failover method, with EstimatedReturnTime and the Maintenance / NoData ServiceLevel sub-ranges.
  • Non-transparent peers resolve through FindServers (ConfiguredRedundantServerSetProvider) and advertise the NTRS discovery capability for GDS/NTRS registration.
  • Deterministic, replica-stable EventId synchronization for Transparent and HotAndMirrored sets (§6.6.2.2), excluding per-replica fields so de-duplicating clients do not double-process events.

Client side (§6.6)

  • DefaultServerRedundancyHandler reads Server.ServerRedundancy and Server.ServiceLevel and fails over to the highest-ServiceLevel peer. It honors Maintenance (Part 4 Table 105 — disconnect to an available peer), keeps Warm backups disabled until failover (Table 107), and treats peers known only through ServerUriArray as failover candidates.
  • RedundantManagedClient implements the Cold, Warm, Hot (a), Hot (b), and HotAndMirrored client patterns: one active session, optional lightweight ServiceLevel status-check sessions to backups, and — for Hot (b) — merging of concurrent reporting streams with exact value-identity de-duplication. HotAndMirrored failover re-activates the mirrored session with the existing AuthenticationToken instead of recreating it.
  • Network redundancy: alternate communication paths via endpoint selection.

Distributed state (opt-in; an extension beyond §6.6)

  • ISharedKeyValueStore (in-memory default) and INodeStateStore, with an AddressSpaceSynchronizer that bridges a CustomNodeManager2's predefined nodes to the shared store, so node and reference additions/removals and value changes propagate to other replicas. IDistributedValueCache lets read/write callbacks cache the last value with a freshness bound; monitored items use the normal read pipeline and therefore participate only when the read path does.
  • Secure session sharing (active/passive fast reconnect), server and client. DistributedSessionManager (wired through a new additive ISessionManagerFactory seam on StandardServer) mirrors encrypted session state across replicas. On a failover reconnect the standby restores the session and performs the full ActivateSession client-certificate signature validation against a single-use serverNonce consumed via compare-and-swap across the replica set, enforcing the same SecurityPolicy and SecurityMode. The AuthenticationToken is a lookup key only, never an authenticator: entries are keyed by the SHA-256 digest of the token, and a cross-replica restore emits a distinct audit event.
  • Subscription, retransmission-queue, and continuation-point mirroring for subscription transfer and failover.
  • Leader election: a static single-leader provider and a shared-store lease via compare-and-swap, supporting active/passive and active/active (shared-read / master-write).
  • Fail-closed security defaults: shared records are protected at rest through IRecordProtector; secret-bearing mirrors require a registered protector or an explicit opt-out.

Active/active (CRDT)

Opc.Ua.Server.Redundancy.Crdt adds leaderless multi-writer address-space replication with CRDTs and gossip (UseReplicatedAddressSpace) plus CRDT-backed session metadata (UseReplicatedSessions). Networked gossip is fail-closed without authenticated transport (mutual TLS for TCP; an explicit opt-out for isolated dev/test). CRDT state is eventually consistent, so exactly-once decisions (such as the single-use nonce registry) stay on a strongly consistent store. The package is available on all stack target frameworks.

Kubernetes

Opc.Ua.Server.Redundancy.K8s provides Kubernetes Lease leader election, EndpointSlice-based peer discovery, and ServiceLevel-driven readiness for running an OPC UA server replica set. Docs/HighAvailabilityKubernetes.md covers StatefulSet/Deployment and Service manifests, RBAC, probes, time synchronization, secrets, gossip-port NetworkPolicy, and GDS/NTRS registration.

Compatibility and validation

  • Zero-overhead single-instance default; additive, backward-compatible seams (for example ISessionManagerFactory and IServerStartupTask); replaced APIs marked [Obsolete] for migration.
  • Builds across the full stack TFM set (net472, net48, netstandard2.0, netstandard2.1, net8.0, net9.0, net10.0); NativeAOT-compatible where the runtime supports it.
  • Unit and integration tests for server, client, subscriptions, session mirroring, CRDT, and Kubernetes; the conformance suite stays green.

See Docs/HighAvailability.md for the full OPC 10000-4 §6.6 mapping, the Add* (standard nodes) versus Use* (extension) builder convention, and the security/rotation guidance.


Update — package consolidation + strong consistency (Raft)

Since the summary above was written, the redundancy packages were consolidated and renamed, client redundancy was folded into the managed session, and a strongly-consistent (Raft) layer was added. The branch is also merged up to the latest master.

Packages (current): Opc.Ua.Redundancy (shared CRDT + Raft building blocks), Opc.Ua.Redundancy.Server, Opc.Ua.Redundancy.Client, Opc.Ua.Redundancy.K8s. (The former *.Server.Redundancy / .Redundancy.Crdt / .Redundancy.K8s names are replaced; the CRDT active/active code merged into the server package.)

Client: RedundantManagedClient is removed — redundancy is transparent and built into ManagedSession (via IManagedSession / ManagedSessionBuilder), so connecting to a non-redundant, redundant (transparent or non-transparent), or client-replica-set server needs no special API.

Strong consistency (Raft), an extension beyond §6.6. A CP Raft layer complements the AP CRDT store behind the same ISharedKeyValueStore seam, selectable with RedundancyConsistencyMode (Eventual default, Strong) via UseRedundancyConsistency:

  • RaftSharedKeyValueStore provides a real linearizable CompareAndSwapAsync + WatchAsync; RaftLeaderElection gives native single-leader election (no split-brain); HybridSharedKeyValueStore routes the strong keyspaces (single-use nonce, lease, election) to Raft and bulk state to the CRDT store.
  • The single-use session nonce registry and lease election become linearizable with no external store (no Redis required).
  • The consensus engine is pluggable via IRaftConsensus: InProcessRaftConsensus for single-process/tests, and the external RaftCs engine (marcschier/raft-cs, over NanoMsg with an optional file WAL) for multi-pod via RaftCsConsensus.CreateCluster / UseKubernetesRaftConsensus. RaftCs is NativeAOT-clean.
  • The RedundantServer sample gains HA_CONSISTENCY=strong and docker-compose.raft.yml (a real cross-container Raft cluster). See Docs/HighAvailability.md (Consistency modes).

…ng with hardened shared store

Adds an opt-in provider model (Opc.Ua.Server.Distributed) to replicate address-space topology/values and session state across server replicas for active/passive and active/active HA, exposed via documented OPC UA redundancy (ServiceLevel/RedundantServerArray). The single-instance in-memory path stays zero-overhead (default NullRecordProtector).

Building blocks: ISharedKeyValueStore (+in-memory), INodeStateStore, address-space synchronizer, leader election (static + shared-store lease CAS), service-level providers, distributed value cache, shared session store. Wired via IServerStartupTask hosting seam + fluent DI (UseDistributedAddressSpace / AddServerServiceLevel / redundancy options). CustomNodeManager2 opts in via ILocalAddressSpaceSource.

Security hardening (plan 30, SDL + IEC 62443): IRecordProtector + AesCbcHmacRecordProtector (AES-256-CBC Encrypt-then-MAC, verify-before-decrypt, fail-closed) on every shared record; KeyRingRecordProtector (staged key rotation); SharedSingleUseNonceRegistry (cross-replica single-use server nonce via store CAS); key zeroization. Docs/HighAvailabilitySecurity.md captures the STRIDE threat model and operator guidance.

Docs: HighAvailability.md, HighAvailabilitySecurity.md, Docs/README link, HighAvailabilityServer sample. Plans 28/29/30. 85 Distributed unit tests (net10.0 + net48).
…reconnect (S5)

Implements the opt-in mirrored fast-reconnect from plans 30/31. After a failover a client reconnects to a standby by re-running ActivateSession; the AuthenticationToken is only a lookup key and the standby still performs the full client-certificate signature validation against a mirrored, single-use serverNonce (token-only hijack and nonce replay are both closed).

- SharedSessionEntry: extended with the full reconstruction state (serverNonce, clientNonce, client cert blob, SecurityPolicy/Mode, endpoint, timeout, client description); encrypted at rest via the record protector.
- SessionManager: additive, backward-compatible RestoreSessionAsync + SupportsSessionRestore seam in the ActivateSession miss-path (default returns null => unchanged behaviour; m_sessions stays private).
- DistributedSessionManager: mirrors encrypted session state on create/activate, removes on close; on restore enforces REQ-UA-7 (same SecurityPolicy/Mode), consumes the serverNonce single-use across the replica set (replay defence), reconstructs the session, and logs provenance with a one-way token digest.
- ISessionManagerFactory seam on StandardServer (wired from DI by the hosted service, supplying a server-certificate provider) + DistributedSessionManagerFactory + UseDistributedSessions(...) fluent API. Safe default EnableFastReconnect=false (re-auth on failover).
- Docs (HighAvailability.md, HighAvailabilitySecurity.md) updated.

Tests: 97 Distributed (net10+net48); no regression (61 Session + 57 client SessionTests integration pass). Remaining: two-server network e2e (S6).
…roring (S6)

DistributedSessionMirrorIntegrationTests stands up a fully-started server whose ISessionManagerFactory builds a DistributedSessionManager, and verifies end-to-end that a session created/activated through the real service handlers is mirrored encrypted to the shared store (a wrong-key reader fails closed) and removed on close. This closes the factory -> StandardServer.CreateSessionManager -> mirror runtime-wiring gap.

The full secured two-server token-reuse reconnect happy-path remains a documented follow-up (the stack client does re-auth-on-failover; the direct-service helper only drives unsecured sessions). The restore decision logic (REQ-UA-7 + single-use nonce/replay) is unit-tested and the base ActivateSession signature path is integration-tested.

98 Distributed tests pass (net10 + net48).
@codecov

codecov Bot commented Jun 26, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0.29630% with 673 lines in your changes missing coverage. Please review.
✅ Project coverage is 20.80%. Comparing base (f9a8343) to head (4e71f45).

Files with missing lines Patch % Lines
...ssion/Redundancy/DefaultServerRedundancyHandler.cs 0.00% 211 Missing ⚠️
...dundancy/DefaultRedundantServerEndpointResolver.cs 0.00% 96 Missing ⚠️
...nt/Session/Redundancy/ClientFailoverCoordinator.cs 0.00% 79 Missing ⚠️
...ent/Session/Redundancy/ClientReplicaCoordinator.cs 0.00% 75 Missing ⚠️
Libraries/Opc.Ua.Client/Session/Session.cs 0.00% 73 Missing ⚠️
...ries/Opc.Ua.Client/Fluent/ManagedSessionBuilder.cs 0.00% 40 Missing ⚠️
...on/Redundancy/NetworkRedundancyEndpointSelector.cs 0.00% 40 Missing ⚠️
Libraries/Opc.Ua.Client/Session/ManagedSession.cs 0.00% 32 Missing ⚠️
...ient/Session/Redundancy/ClientReplicaSetBuilder.cs 0.00% 21 Missing ⚠️
...c.Ua.Client/Fluent/OpcUaClientBuilderExtensions.cs 25.00% 3 Missing ⚠️
... and 3 more

❗ There is a different number of reports uploaded between BASE (f9a8343) and HEAD (4e71f45). Click for more details.

HEAD has 69 uploads less than BASE
Flag BASE (f9a8343) HEAD (4e71f45)
72 3
Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff             @@
##           master    #3918       +/-   ##
===========================================
- Coverage   73.52%   20.80%   -52.73%     
===========================================
  Files        1169      659      -510     
  Lines      169907    92117    -77790     
  Branches    29296    16666    -12630     
===========================================
- Hits       124927    19165   -105762     
- Misses      34005    70285    +36280     
+ Partials    10975     2667     -8308     
Files with missing lines Coverage Δ
...ies/Opc.Ua.Client/Session/ManagedSessionOptions.cs 100.00% <ø> (ø)
...Client/Session/Reconnect/ConnectionStateMachine.cs 0.00% <ø> (ø)
...Opc.Ua.Client/Session/Reconnect/ReconnectPolicy.cs 0.00% <ø> (ø)
...Client/Session/Reconnect/ReconnectPolicyOptions.cs 100.00% <ø> (ø)
...lient/Session/Reconnect/SessionReconnectHandler.cs 0.00% <ø> (ø)
...ent/Session/Redundancy/NetworkRedundancyOptions.cs 100.00% <100.00%> (ø)
.../Session/Subscription/ClassicSubscriptionEngine.cs 0.00% <ø> (ø)
...n/Subscription/ClassicSubscriptionEngineFactory.cs 0.00% <ø> (ø)
.../Session/Subscription/DefaultSubscriptionEngine.cs 0.00% <ø> (ø)
...n/Subscription/DefaultSubscriptionEngineFactory.cs 0.00% <ø> (ø)
... and 22 more

... and 1020 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…t (F6/F9)

Workstream A from plans/32:
- F6: SharedKeyValueSessionStore keys entries by the SHA-256 digest of the AuthenticationToken (SharedKeyValueSessionStore.KeyFor) instead of the raw token, so a backend's key enumeration/monitoring/dumps never expose the token. Test asserts the keyspace contains no raw token.
- F9: a successful cross-replica restore emits a distinct AuditSessionEventState (Session/RestoredFromSharedStore) via the new IAuditEventServer.ReportAuditSessionRestoredEvent, in addition to the standard AuditActivateSession, with a one-way token digest for provenance.
- F7: analyzed — the decrypted serverNonce becomes the session's retained working Nonce (no extra plaintext copy in the manager); Nonce.Data zeroization on dispose is a pre-existing server-wide Core concern tracked separately.

Docs updated. 99 Distributed tests pass (net10 + net48).
…ng in sample (FG)

Workstream FG from plans/32:
- New Docs/KubernetesDeployment.md: worked replicaset deployment (StatefulSet + headless Service, leader election, readiness tied to ServiceLevel, KEK + shared ApplicationInstanceCertificate provisioning via Secrets, security checklist). Linked from Docs/README.md and HighAvailability.md.
- HighAvailabilityServer sample: wires UseDistributedSessions (opt-in HA_FAST_RECONNECT) and an optional AesCbcHmacRecordProtector from a base64 HA_RECORD_KEY (encrypted shared store), demonstrating the production-hygiene pattern.
- Transparent redundancy remains a documented deployment pattern (single virtual endpoint + shared session store + subscription transfer; no new transport).
…A-13, B)

On failover to a redundant server, when EnableTokenReuseFailover is set, the client re-activates the existing session by reusing the current AuthenticationToken (signing over the new channel + last serverNonce) instead of CreateSession, falling back to re-authentication if the standby rejects the token.

- Session.cs: extracted ReactivateExistingSessionAsync (the token-reuse activation core, shared with UpdateSessionAsync); RecreateInPlaceCoreAsync tries it first (adopting the failover server's cert) before the existing clear + fresh-CreateSession fallback; new EnableTokenReuseFailover property (copied across recreate clones). OpenAsync is untouched.
- ManagedSession: EnableTokenReuseFailover option threaded through CreateAsync + ctor and applied to the inner session; ManagedSessionOptions.EnableTokenReuseFailover; ManagedSessionBuilder.WithTokenReuseFailover(). Default off (re-auth on failover).

No regression: 57 client SessionTests + 135 reconnect/failover integration tests pass (net10). Updated the ctor-reflection test for the new parameter.
DistributedSessionFailoverIntegrationTests: two secured servers share one store via DistributedSessionManager; a ManagedSession client with WithTokenReuseFailover fails over from the active to the standby. The standby restores the mirrored session and re-activates it with the reused AuthenticationToken, so the client's SessionId is preserved (a fresh re-authentication would change it). Passes on net10 + net48.

Docs (HighAvailability.md, HighAvailabilitySecurity.md) updated: client WithTokenReuseFailover() usage + the e2e validation; plans/32 marks B and C done.
Comment thread Applications/HighAvailabilityServer/HaSampleNodeManager.cs Outdated
Comment thread Applications/HighAvailabilityServer/README.md Outdated
Comment thread Libraries/Opc.Ua.Client/Fluent/ManagedSessionBuilder.cs Outdated
Comment thread Stack/Opc.Ua.Core/Redundancy/AesCbcHmacRecordProtector.cs
Comment thread Libraries/Opc.Ua.Server/Distributed/AesCbcHmacRecordProtector.cs Outdated
Comment thread Libraries/Opc.Ua.Server/NodeManager/CustomNodeManager.cs
Comment thread Libraries/Opc.Ua.Server/NodeManager/ILocalAddressSpace.cs
Comment thread Docs/HighAvailabilitySecurity.md Outdated
Split all distributed/HA implementation into a new
OPCFoundation.NetStandard.Opc.Ua.Server.Distributed project that references
Opc.Ua.Server; core now keeps only the seams.

- Move ILocalAddressSpace/ILocalAddressSpaceSource to the Opc.Ua.Server
  namespace (NodeManager/); extract a shared internal PredefinedNodesAddressSpace
  reused by both CustomNodeManager2 and AsyncCustomNodeManager (both now
  implement ILocalAddressSpaceSource). Apply path is async (no sync-over-async).
- Remove the node-state-store-registry hook from ServerInternalData; the
  distributed startup task owns its own registry.
- Add CryptoUtils.ZeroMemory/FixedTimeEquals polyfills and reuse them in the
  record protector, EncryptedSecret, and KeyCredentialBridgeAuthenticator.
- Fix CA2213 in AddressSpaceSynchronizer (dispose enumerator in DisposeAsync).
- Rebase HighAvailabilityServer sample onto AsyncCustomNodeManager; expand the
  sample README with shared-store wiring and an active/active setup.
- Strip internal tracking tags (REQ-UA/Finding/IEC/SDL) from code and XML docs.
- Merge HighAvailabilitySecurity.md into HighAvailability.md and fix links.
- Wire the new project into UA.slnx, the test project, and the sample.
Add a separate net8.0+ package providing true active/active (multi-writer)
replication for the distributed server, built on the Crdt + Crdt.Transport
NuGet packages. Opt-in; the base distributed library and its active/passive
path are unchanged.

- New Libraries/Opc.Ua.Server.Distributed.Crdt project (net8.0;net9.0;net10.0,
  no-op shell on legacy CI legs via RestrictForLegacyTfm); references the base
  distributed project + Crdt + Crdt.Transport.
- Address space A/A: CrdtAddressSpaceSynchronizer models topology and values as
  last-writer-wins maps and gossips state over Crdt.Transport (in-memory / TCP /
  UDP, optional TLS); every replica writes, received state is merged and applied.
  A topology merge preserves the locally-known value so it never regresses a
  concurrently-updated value (values are versioned by their own entries).
- Sessions A/A: CrdtSharedKeyValueStore replicates encrypted session entries by
  gossip and reuses DistributedSessionManager; the single-use server nonce stays
  on a strongly-consistent ISingleUseNonceRegistry (CRDTs cannot enforce
  exactly-once), and the CRDT store rejects compare-and-swap.
- Fluent opt-in: UseCrdtAddressSpace(...) / UseCrdtSessions(...) with a shared
  CrdtGossipOptions base (ReplicaId, UseTcpGossip/UseUdpGossip/AddPeer, limits).
- Tests: new Opc.Ua.Server.Distributed.Crdt.Tests (net8/net10) — convergence,
  multi-writer, concurrent-value LWW, serializer round-trip, CAS/Watch boundary;
  plus a CRDT exercise in the AOT test project.
- Docs: HighAvailability.md 'Active/active with CRDTs' section incl. the
  single-use-nonce security boundary; Crdt/Crdt.Transport added to CPM.

Crdt/Crdt.Transport 1.0.0 (MIT, NativeAOT-ready) restore from nuget.org.
Comment thread Docs/HighAvailability.md Outdated
Comment thread Docs/HighAvailability.md Outdated
Comment thread Docs/HighAvailability.md Outdated
Comment thread Docs/HighAvailability.md Outdated
Comment thread Applications/HighAvailabilityServer/README.md Outdated
Comment thread Libraries/Opc.Ua.Redundancy.Server/Opc.Ua.Redundancy.Server.csproj
…mple, clarify HA docs

- Rename the user-facing fluent surface from Crdt* to Replicated*:
  UseReplicatedAddressSpace / UseReplicatedSessions, ReplicatedAddressSpaceOptions
  / ReplicatedSessionOptions / ReplicatedGossipOptions, ReplicatedServerBuilderExtensions
  (implementation classes and the Crdt package/namespace keep their names).
- Integrate the CRDT package into the HighAvailabilityServer sample with an
  HA_MODE switch (ap = active/passive leader-write, aa = active/active CRDT
  gossip via HA_GOSSIP_PORT / HA_GOSSIP_PEERS); document running two AA replicas
  in the sample README.
- HighAvailability.md: reference the implemented CRDT library (not 'deferred'),
  document that a write to a non-leader replica is discarded in active/passive,
  and rewrite the value-participation note to describe how participation works.

Addresses review comments on Docs/HighAvailability.md and the sample README.
…Ua.Server.Distributed.Tests

Organize the source files of both Opc.Ua.Server.Distributed and
Opc.Ua.Server.Distributed.Crdt into logical subfolders (AddressSpace,
KeyValueStore, Redundancy, Values, Sessions, Security for the base library;
AddressSpace, Sessions for the CRDT adapter), with the top-level DI entry
points kept at the project root. Namespaces are unchanged.

Mirror the organization in the test projects:
- Extract the base distributed tests from Opc.Ua.Server.Tests/Distributed into a
  dedicated Opc.Ua.Server.Distributed.Tests project (mirroring the src project),
  with the same subfolders; add it to UA.slnx and InternalsVisibleTo. Test
  namespaces are unchanged.
- Organize Opc.Ua.Server.Distributed.Crdt.Tests into AddressSpace/Sessions
  subfolders mirroring the CRDT src.

Addresses review comments asking to mirror the test projects to the src side and
to organize the files logically without changing namespaces.
@marcschier marcschier changed the title [Server] Distributed high-availability: shared address space + secure session sharing [Server] Distributed high-availability: shared address space, secure sessions + active/active (CRDT) Jun 26, 2026
Add unit tests for the previously-uncovered CRDT adapter surface flagged by codecov (patch coverage 67.65%):

- ReplicatedGossipOptions/ReplicatedSessionOptions: defaults, UseTcp/UseUdp transport factory wiring, AddPeer, CreateReaderOptions/CreateTransport, null-arg guards.

- CrdtAddressSpaceStartupTask: attaches a synchronizer to opted-in (ILocalAddressSpaceSource) node managers and skips the rest; null-arg guards.

- CrdtSessionManagerFactory + UseReplicatedSessions/UseReplicatedAddressSpace registration; null-arg guards.

- ByteStringCrdtSerializer JSON round-trip (null/empty/data); CrdtSharedKeyValueStore.ScanAsync prefix filtering.

CRDT package line coverage 67% -> 93.7%; 25/25 tests pass on net8.0 and net10.0.
@marcschier

Copy link
Copy Markdown
Collaborator Author

Addressed the codecov patch-coverage feedback in 4f4d7ed.

The 80% patch gap was concentrated in the new Opc.Ua.Server.Distributed.Crdt adapter (the ReplicatedGossipOptions, CrdtAddressSpaceStartupTask, and CrdtSessionManagerFactory files were at ~0%). Added unit tests covering:

  • ReplicatedGossipOptions/ReplicatedSessionOptions — defaults, UseTcpGossip/UseUdpGossip transport-factory wiring, AddPeer, CreateReaderOptions/CreateTransport, and null-arg guards.
  • CrdtAddressSpaceStartupTask — attaches a synchronizer to opted-in (ILocalAddressSpaceSource) node managers and skips the rest.
  • CrdtSessionManagerFactory + the UseReplicatedSessions/UseReplicatedAddressSpace registrations.
  • ByteStringCrdtSerializer JSON round-trip (null/empty/data) and CrdtSharedKeyValueStore.ScanAsync prefix filtering.

CRDT package line coverage rises from ~67% to 93.7%; 25/25 tests pass on net8.0 and net10.0. Codecov will re-evaluate the patch on this commit.

…t + non-transparent

Server model: per-mode Server.ServerRedundancy (RedundancySupport, RedundantServerArray, non-transparent ServerUriArray, transparent CurrentServerId); sub-range ServiceLevel providers (Table 105) + load balancing; RequestServerStateChange/Maintenance/EstimatedReturnTime (6.6.5, admin-gated); NTRS capability; FindServers returns the RedundantServerSet (IRedundantServerSetProvider). New shared Opc.Ua.ServiceLevels/ServiceLevelSubrange in core.

Client: RedundantManagedClient realizing all Table 107 modes (Cold/Warm/Hot(a)/Hot(b)/HotAndMirrored); RedundancySupport wording; ServiceLevel sub-range failover rules; FindServers ServerUri->endpoint resolution (no security downgrade); client redundancy via TransferSubscriptions; non-transparent network redundancy.

State mirroring (opt-in provider seams; single-instance unchanged): session takeover, subscription-definition mirror, async notification sequence/Republish mirror, best-effort continuation-point envelope, deterministic EventId provider, RegisterNodes replica-consistent.

Extensions: Opc.Ua.Server.Distributed.Crdt (active/active) and Opc.Ua.Server.Distributed.Kubernetes (Lease election, peer discovery, ServiceLevel->readiness). Samples (HighAvailabilityServer, RedundantClient), docs mapped to 6.6, conformance tests.

Security: AES-CBC+HMAC record protection, fail-closed RecordProtectionGuard for external stores (base + CRDT session paths), single-use server-nonce replay protection, K8s TLS hostname validation. Validated on net10 and net48.
… review findings

CRDT active/active extension (Opc.Ua.Server.Distributed.Crdt):
- Bump Crdt/Crdt.Transport 1.0.0 -> 1.0.2 (now ship netstandard2.0/2.1 assets)
- Widen TFMs from net8.0+ to $(LibTargetFrameworks) (net472/net48/netstandard2.1/net8/9/10);
  remove RestrictForLegacyTfm; gate IsAotCompatible/binding-gen to net8+
- Fail closed for unauthenticated networked gossip (TCP without mutual TLS, UDP); add
  AllowUnauthenticatedGossip opt-out for dev/test; gossip-port NetworkPolicy guidance

Server review findings:
- AddServerRedundancy warns when non-transparent mode has no IServiceLevelProvider
- Rename AddManualFailover -> AddRequestServerStateChange ([Obsolete] shim retained)
- Consolidate ServerRedundancyOptions peer inputs (RedundantPeers canonical)
- DeterministicEventId: exclude per-replica ReceiveTime, compute after distinguishing
  fields are populated (replica-stable per 6.6.2.2)
- Subscription retransmission mirror: delta path, namespace/server tables once per
  subscription, bounded-parallel drain

Client review findings:
- Maintenance(0) fails over to a healthy peer (Table 105); EstimatedReturnTime backoff
- Warm backups use MonitoringMode.Disabled until failover (Table 107)
- Hot(b) dedup uses exact value identity (no hash-collision drops); value-struct key
- Subscription-template ownership: client disposes retained templates
- De-duplicate WithNetworkRedundancy registration

Validated: 0 warnings; tests green on net10 + net48 (CRDT 30, Distributed 178,
Client redundancy 75, InformationModel redundancy 7).
…eview feedback

Rename (full identity: folders, csproj, AssemblyName, PackageId, RootNamespace,
C# namespaces, references, UA.slnx, InternalsVisibleTo):
- Opc.Ua.Server.Distributed        -> Opc.Ua.Server.Redundancy
- Opc.Ua.Server.Distributed.Crdt   -> Opc.Ua.Server.Redundancy.Crdt
- Opc.Ua.Server.Distributed.Kubernetes -> Opc.Ua.Server.Redundancy.K8s (identity only;
  the word "Kubernetes" and k8s client types are preserved in prose/code)
- Tests mirror the rename (.Tests/.Crdt.Tests/.K8s.Tests/.Integration.Tests)
- Application HighAvailabilityServer -> RedundantServer
- Documentation updated to the new names (present tense, merged-to-master state)

Review feedback (verified previously-resolved PR comments are satisfied; fixed gaps):
- async node-manager base, A/A sample README, REQ-UA tags, CA2213, central
  CryptoUtils polyfills, ILocalAddressSpace namespace - all confirmed in current code

Roadmap findings implemented:
- OPCFoundation#28 ServerUriArray-only peers are now failover candidates (connect + read live level)
- OPCFoundation#8 FetchRedundancyInfo follow-up reads merged into one ReadValuesAsync
- OPCFoundation#29 ContinuationPoint mirroring documented as envelope-only (partial-SHALL boundary)
- OPCFoundation#30 Server.ServerRedundancy typed to NonTransparent/Transparent subtype by mode
- OPCFoundation#20 Add* (standard) vs Use* (extension) convention documented + ServiceLevel cross-ref
- OPCFoundation#24 transparent-mode shared application-key blast-radius/rotation guidance expanded

CI green (fixes pre-existing all-TFM build break, unrelated to the rename):
- Opc.Ua.Sessions.Tests redundancy test doubles: implement IServerRedundancyHandler.
  ShouldFailover and use RedundancySupport (RedundancyMode was removed)
- Integration tests: add RestrictForLegacyTfm + CustomTestTarget fallback so legacy
  TFM CI passes no-op instead of failing with empty TargetFrameworks
- Integration Maintenance assertion updated to spec-aligned behavior (Maintenance with
  an available peer warrants failover, Part 4 Table 105)

Validated: full UA.slnx builds on net10 + net48; Redundancy 178, Crdt 30, K8s 27,
Integration 3, Client redundancy 76, Sessions failover 4 - all pass.
The private ManagedSession constructor gained enableTokenReuseFailover (bool) and
networkRedundancy (NetworkRedundancyOptions?) parameters during the network-redundancy
work, but two reflection-based test helpers still used the old signature, so
GetConstructor returned null and failed the fixture SetUp (52 cascading failures =
the test-ubuntu-latest-Client / Fast PR test CI failures).

- ManagedSessionComplianceTests.CreateManagedSessionWithInner: add the 4th bool +
  NetworkRedundancyOptions to the ctor type list and invoke args
- ManagedSessionTests: add NetworkRedundancyOptions to the ctor type list and invoke args

Validated: full Opc.Ua.Client.Tests now 1530 passed / 0 failed on net10.
The legacy-TFM no-op shell strips all references, so on netstandard2.0/2.1 there is no
assembly providing System.Object and the empty compile failed with CS8021 ("No value
for RuntimeMetadataVersion"). net4x builds got System.Object from the implicit targeting
pack so they worked. Supply an explicit RuntimeMetadataVersion for the no-op shell so the
empty assembly compiles on every legacy TFM.

This was the netstandard2.0 leg of the build-*-all-tfm failure; it affected every
RestrictForLegacyTfm project (Core.Diagnostics, Network.Fuzz*, Redundancy.K8s*,
Redundancy.Integration.Tests, RedundantServer, RedundantClient, Mcp, Minimal* samples).

Validated: full UA.slnx builds on net10, net48, and netstandard2.0; the no-op shell
compiles cleanly on net472/net48/netstandard2.0/netstandard2.1.
@marcschier marcschier changed the title [Server] Distributed high-availability: shared address space, secure sessions + active/active (CRDT) OPC UA Part 4 6.6 Redundancy (server + client) with opt-in distributed high-availability Jun 27, 2026
… from codecov

The new redundancy libraries are ~80% unit-covered, but codecov/patch was dragged
down by genuinely-untestable Kubernetes IO (real HTTP to the API server, the
readiness HTTP listener, and cluster-bound startup tasks).

- Add KubernetesServerBuilderExtensionsTests: a DI-registration test that exercises
  UseKubernetes / UseKubernetesLeaderElection / UseKubernetesPeerDiscovery /
  UseKubernetesReadiness (the previously 0%-covered builder wiring) plus null-builder
  guards, resolving the registered services out-of-cluster with no IO.
- codecov.yml: ignore the four integration-only K8s IO files (KubernetesHttpApiClient,
  KubernetesReadinessServer, KubernetesReadinessStartupTask,
  KubernetesPeerDiscoveryStartupTask) - mirrors the existing Applications/** exemption;
  the Kubernetes logic (models, factory, lease election, peer parsing, readiness
  mapping, builder wiring) stays measured.

Measured redundancy-library line coverage (codecov view) is now ~88%, above the 80%
patch target. K8s tests: 29 passed.
…Lock ambiguity)

The net10-only RedundantServer sample used RestrictForLegacyTfm, which only no-ops
legacy TFMs (net4x/ns2.x). Under the Linux all-TFM CI leg (CustomTestTarget=net8.0)
the app still built net10 while referencing the net8-built Opc.Ua.Types, so its
System.Threading.Lock polyfill collided with net10's BCL Lock (CS0433) in
HaSampleNodeManager. Follow CustomTestTarget like the libraries so the app's framework
references match the TFM being built (net8 app + net8 types => only the polyfill Lock).

Validated: RedundantServer builds on net8.0 and net10.0.
Core.Diagnostics.Tests builds net8/net9/net10 and had an unconditional build-ordering
ProjectReference to the net10-only McpServer (so its assembly exists for the reflective
McpServer tests). On the Linux all-TFM CI legs (CustomTestTarget=net8.0/net9.0) the
solution build forced McpServer to the leg TFM, which it cannot target (its own deps
build net8/net9), failing the UA.slnx build.

- Reference McpServer only when building net10 (CustomTestTarget '' or net10.0).
- Make both reflective LoadMcpAssembly helpers Assert.Ignore (skip) when the net10
  Opc.Ua.Mcp assembly is absent, instead of asserting it exists - so the net8/net9 test
  legs skip the MCP reflective tests cleanly while net10 still runs them.

Pre-existing infra issue (unrelated to the redundancy feature) surfaced once the
RedundantServer net8 Lock fix let the Linux all-TFM build progress past net8.

Validated: full UA.slnx builds on net8.0/net9.0/net10.0; Core.Diagnostics McpServer
tests pass on net10 (13) and skip/pass without failure on net8.
Comment thread Docs/MigrationGuide.md Outdated
Comment thread Docs/KubernetesDeployment.md Outdated
Comment thread Docs/HighAvailability.md Outdated
Merge Opc.Ua.Redundancy.Server.Crdt.Tests into Opc.Ua.Redundancy.Server.Tests (208 tests, 39s, well under 20min); remove the empty Opc.Ua.Redundancy.Server.Integration.Tests project (no integration tests exist); rename Opc.Ua.Client.Redundancy.Tests to Opc.Ua.Redundancy.Client.Tests. Full UA.slnx builds net10; Server 208, Client 6 pass.
New Opc.Ua.Redundancy library hosts the canonical ByteStringCrdtSerializer + CrdtSharedKeyValueStore (namespace Opc.Ua.Redundancy, shared with the Core seams). Opc.Ua.Redundancy.Server and .Client both link it; their duplicate CRDT copies removed. Adds AddCrdtClientSharedStore DI extension in Opc.Ua.Redundancy.Client (server CRDT DI already via ReplicatedServerBuilderExtensions). Full UA.slnx builds net10+net48; Server 208, Client 6 tests pass.
…phase 1)

Adds the strongly-consistent consensus seam IRaftConsensus (modeled 1:1 on the RaftNode facade of marcschier/raft-cs) plus a deterministic in-process backend (InProcessRaftConsensus + InProcessRaftCluster) with a single shared totally-ordered committed log and lowest-live-id leadership. This is the offline/test backend behind the upcoming RaftSharedKeyValueStore and RaftLeaderElection; the external multi-node RaftCs engine binds to the same contract later.
…erElection (Raft phase 2)

RaftSharedKeyValueStore is a replicated state machine over IRaftConsensus: Set/Delete/CAS commands are proposed, applied in committed log order against a materialized map (CAS decided deterministically at apply time), and correlated back to the caller by request id. Provides a real linearizable CompareAndSwapAsync and a WatchAsync change-feed (derived from the commit stream), plus TryGet/Scan from the materialized snapshot. RaftLeaderElection delegates ILeaderElection to native Raft leadership (no lease CAS, no split-brain). 19 unit tests incl. exactly-one-CAS-winner under contention and two-replica convergence over a shared cluster.
…cyMode (Raft phase 3)

RedundancyConsistencyMode { Eventual (default) | Strong }. HybridSharedKeyValueStore implements the Eventual mode: each key lives in exactly one backend by prefix - strong prefixes (nonce/, lease/, election/, configurable via ArrayOf<string>) route entirely to the linearizable Raft store; all other keys route to the eventually-consistent CRDT store. A spanning/empty-prefix scan merges both. 8 routing unit tests.
…onsistency (Raft phase 4)

Adds RedundancyConsistencyOptions + UseRedundancyConsistency(builder) which registers a shared IRaftConsensus singleton, the mode-appropriate ISharedKeyValueStore (Strong -> RaftSharedKeyValueStore; Eventual -> HybridSharedKeyValueStore over a CRDT/InMemory bulk + Raft strong store), and a native RaftLeaderElection. Uses TryAddSingleton so it composes before UseDistributedAddressSpace/UseDistributedSessions and wins. 4 DI tests.
SharedSingleUseNonceRegistry and SharedStoreLeaseElection already consume ISharedKeyValueStore.CompareAndSwapAsync, so registering a Raft/Hybrid store (phase 4) makes them linearizable with no code change. Tests prove: exactly-once nonce under contention over Raft; single-leader lease under contention; and that UseRedundancyConsistency auto-wires a CAS-capable store for the CRDT session factory's nonce registry (no separate Redis backend required).
…re (Raft phase 6)

AddRaftClientSharedStore registers a Raft-backed ISharedKeyValueStore + native RaftLeaderElection sharing one IRaftConsensus, giving a client replica set linearizable shared state and primary-client election in one call (the ClientReplicaCoordinator consumes both). AddRedundantClientSharedStore(mode,...) is mode-aware: Strong -> Raft store; Eventual -> Hybrid over a CRDT bulk + Raft strong store. 4 client DI tests.
…s (Raft phase 7, gated)

Adds the RaftCsConsensus : IRaftConsensus adapter that wraps a RaftCs RaftNode (ProposeAsync, Committed apply-stream, IsLeader -> LeadershipChanged poll). Compiled out behind #if OPCUA_RAFTCS so the repo builds/tests fully offline against InProcessRaftConsensus. The file header documents the activation steps (add RaftCs/RaftCs.Transport[.NanoMsg] package refs from Crdt 1.0.5, define OPCUA_RAFTCS, wire via RaftConsensusFactory). No package references added yet - deferred until the user signals the nuget.org pull.
HighAvailability.md gains a 'Consistency modes' section (Strong Raft vs Eventual CRDT-complemented-by-Raft, UseRedundancyConsistency, RaftLeaderElection, client extensions, RaftCs/k8s notes) and the stale 'keep nonce on a strongly consistent store (e.g. Redis)' caveat now points at the in-package Raft layer. HighAvailabilityKubernetes.md notes the native Raft election + StatefulSet/quorum/WAL guidance. NugetREADME for Opc.Ua.Redundancy lists the CP/AP/Hybrid building blocks.
…ft consensus

Bumps Crdt/Crdt.Transport 1.0.2 -> 1.1.0 and adds RaftCs, RaftCs.Transport (+ .Transport.NanoMsg, .Storage.File) 1.1.0 to central package management. Opc.Ua.Redundancy now references RaftCs + RaftCs.Transport and defines OPCUA_RAFTCS, activating the RaftCsConsensus adapter (which compiled 1:1 against the real RaftNode API). Adds RaftCsConsensus.CreateSingleNode() (a real single-voter RaftCs replica with in-memory storage/transport that self-elects) plus a bounded wait-for-initial-leader so the first proposals are not dropped during election, and owned-host disposal for the in-memory network.

Wires RaftCs as the DI default IRaftConsensus in UseRedundancyConsistency (server) and AddRaftClientSharedStore/AddRedundantClientSharedStore (client); InProcessRaftConsensus remains the lightweight deterministic default for the parameterless store ctor and direct/unit use, and RaftConsensusFactory plugs in a multi-node RaftNode.

Validation: Server.Tests 246/246 + Client.Tests 10/10 on net10 AND net48; new RaftCsConsensusTests exercise election/CAS/nonce over the real engine; a NativeAOT publish of Opc.Ua.Aot.Tests runs 2 new RaftAotTests green (RaftCs is trim/AOT-clean). Docs/HighAvailability.md + NugetREADME updated to reflect the activated default.
…fallback (review)

Code-review follow-ups (security + data-consistency were clean):

HIGH: RaftSharedKeyValueStore proposals could hang forever in a multi-pod no-leader / leadership-change / lost-quorum window (RaftCs drops a follower proposal when LeaderId==0, and the commit signal is only the Committed stream). Add a bounded commit timeout (default 30s, configurable; Timeout.InfiniteTimeSpan to opt out) that fails the pending proposal with TimeoutException, and fail all pending proposals if the single applier ever dies. The single-node default was already safe.

MEDIUM: In Eventual mode the distributed address-space change-feed routed to a CRDT bulk store whose WatchAsync throws NotSupported, silently faulting the standby's topology-sync task. InMemoryNodeStateStore.SubscribeChangesAsync now catches NotSupportedException and falls back to periodic scan-polling (configurable interval, default 2s) so standby sync keeps working over any store.

LOW: the UseRedundancyConsistency-before-UseDistributed* ordering requirement remains documented (TryAddSingleton). Tests: ProposalTimesOutWhenNoCommitOccurs + SubscribePollsWhenStoreHasNoWatch; full Server.Tests 248/248 on net10 AND net48.
…y + 3-node tests (rw-A1/B)

CreateCluster builds a multi-node RaftCs replica over static ConfState membership with an injectable IRaftTransport + IRaftWritableStorage (so Opc.Ua.Redundancy stays NanoMsg/File-free and AOT-clean); a memberIds overload uses in-memory MemoryStorage, a storage overload takes a durable WAL, and CreateSingleNode is refactored to a 1-member special case. RaftCsClusterTests spin up a real 3-node cluster over RaftCs InMemoryNetwork: follower-write convergence (leader forwarding), leader failover + re-election stays writable, and quorum-loss makes a proposal fail via the commit timeout. 7 tests pass in ~1s.
… (rw-A2)

Opc.Ua.Redundancy.K8s now references RaftCs.Transport.NanoMsg + RaftCs.Storage.File (AOT-clean spike: builds 0-warning with IsAotCompatible; NanoMsgSharp is managed). UseKubernetesRaftConsensus derives this replica's Raft node id from the StatefulSet ordinal, builds member ids from ReplicaCount (static membership), wires a NanoMsgBusTransport (bind + headless-DNS peers) and a crash-safe FileRaftStorage on a PersistentVolume path (in-memory optional; a fresh WAL is bootstrapped with the static ConfState), and registers the result via RaftCsConsensus.CreateCluster as IRaftConsensus so UseRedundancyConsistency composes over it. 3 registration tests.
…tServer (rw-C)

Adds HA_CONSISTENCY=strong to the RedundantServer sample: it registers UseRedundancyConsistency(Strong) backed by a multi-node RaftCs cluster over NanoMsg (HA_RAFT_ID/MEMBERS/BIND/PEERS), so the shared store, leader election, and session nonce are linearizable and shared across containers - the real cross-container HA the in-memory active/passive compose lacks. Adds docker-compose.raft.yml (a runnable 3-node Raft cluster) and documents HA_CONSISTENCY + HA_RAFT_* and the Raft-vs-Redis alternative in the README. Sample builds AOT-clean (0 warnings).
… multi-node wiring (rw-A3)

HighAvailability.md now points at RaftCsConsensus.CreateCluster + UseKubernetesRaftConsensus (and the RedundantServer HA_CONSISTENCY sample) for multi-pod; HighAvailabilityKubernetes.md documents UseKubernetesRaftConsensus (ordinal->id, headless-DNS peers, File WAL on a PVC) with a code example; the K8s NugetREADME lists the new extension.
Coverage pass: all new Raft library files are >=80% from Server.Tests (RaftLeaderElection 83%, RaftCsConsensus 88%, RaftSharedKeyValueStore 92%, Hybrid 93%, InProcessRaftConsensus 95%, cluster/options/builder 96-100%). Adds a K8s test exercising the previously-untested UseDurableStorage=true branch (FileRaftStorage on a temp dir + fresh-WAL ConfState bootstrap) with fully-qualified peer DNS.
Resolved conflicts: UA.slnx (kept both feature Redundancy projects and upstream PubSub/Core.Schema.Tests projects); CryptoUtils.cs (took upstream canonical ZeroMemory/FixedTimeEquals with the broader NETSTANDARD2_1_OR_GREATER||NET5_0_OR_GREATER guard); ManagedSessionBuilder.cs (combined upstream BuildChannelBindings/WithDefaultTcp + the new WebApi endpoint branch with the feature's ownedHttpClientFactory try/finally disposal safety).
- Session: move browse/history continuation-point bookkeeping (store, lists, mirrored-owner maps) into a focused SessionContinuationPoints class; Session delegates via thin forwarders (review t1).
- Subscription: move sent-message ring, sequence-number counter and retransmission-store plumbing into a focused SentMessageQueue class, keeping Publish/Acknowledge/Republish focused on orchestration (review t2).
- Rename Docs/HighAvailabilityKubernetes.md to Docs/Kubernetes.md and update references (review t3).
- Update SubscriptionLifecycleTests reflection helper to reach m_sentMessages via m_messageQueue.

Behavior preserved (verbatim logic moved); validated: Subscriptions 505, Redundancy.Server 251, Server 1895, Sessions continuation-point 29 pass; Opc.Ua.Server builds clean net10+net48.
…aining-work file

Plans 28-32 were almost entirely delivered (CRDT + Raft in-package, secure session mirroring, transparent/non-transparent redundancy, samples, docs). Replace the five superseded plan files with one plans/28-distributed-ha-remaining.md that lists only the not-yet-delivered work: async ISubscriptionStore, transparent-redundancy worked sample, and large-address-space hydration optimization.
Comment thread Docs/Kubernetes.md
Comment thread Libraries/Opc.Ua.Server/Session/Session.cs Outdated
Comment thread Libraries/Opc.Ua.Server/Subscription/Subscription.cs Outdated
return;
}

// A single ManagedSession is the managed client. WithServerRedundancy() lets it

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove "managed client" wording. Key point: "Create a normal managed session, opt it into Server redundancy handling, thats it"

session.ConnectionStateChanged += OnConnectionStateChanged;

await LogRedundancyInfoAsync(session, ct).ConfigureAwait(false);
await SubscribeToCurrentTimeAsync(session, ct).ConfigureAwait(false);

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should also subscribe to a "Replicated" value (from ha node manager in redundantServer sample).

}
}

private static async Task RunReplicaSetAsync(

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe this can be generalized in Opc.Ua.Client or Opc.Ua.Redundancy.Client to be set up using DI (key store, etc.) and that makes for a simpler setup for users that want to use client redundancy? Ideally the user just gets the shape of ISession to worry about and the redundancy is "transparent" (which hides all coordination and sessions that need to be managed). If the "session" was built with > 1 replicas. Whereby if it was built with 0/1 replicas it just defaults to the current implementation and user gets just a ManagedSession (ISession) just like today. Then above would simplify as the setup of the session and subscriptions are the same.

@@ -0,0 +1,91 @@
# Strongly-consistent (Raft) active/passive redundant OPC UA servers plus a managed client.
#
# Unlike docker-compose.active-passive.yml (whose default in-memory store is private to each

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make this the real active-passive.yml and remove the "existing active/passive yml".

@@ -0,0 +1,63 @@
# Active/active redundant OPC UA servers (CRDT gossip) plus a managed client.
#

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make eventual consistency/strong consistency configurable via env var.

IF possible, make a single docker-compose.yml that is env configurable to run as

  • a/a eventual (default),
  • a/a strong,
  • a/p

IF possible also make the number of replicas configurable

# Active/passive: leader-election wiring demonstration, plus a client.
docker compose -f Applications\RedundantServer\docker-compose.active-passive.yml up --build

# Strong consistency: a real 3-node RaftCs cluster (shared store across containers), plus a client.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add docker-compose.yml to the client as well to run either single, or multi replica clients against the server. Determine a good way to show the replication working, e.g. by scaling up the number of clients to saturate a single replica, etc.

Comment thread Docs/Kubernetes.md

The API names distinguish standardized OPC UA model wiring from deployment extensions. `AddServerRedundancy(...)`, `AddServerServiceLevel(...)`, and `AddRequestServerStateChange(...)` publish or maintain OPC 10000-4 §6.6 nodes/methods; `UseDistributedAddressSpace(...)`, `UseDistributedSessions(...)`, `UseDistributedSubscriptionMirroring(...)`, `UseKubernetesLeaderElection(...)`, `UseKubernetesPeerDiscovery(...)`, and `UseKubernetesReadiness(...)` register beyond-spec extension services. `AddServerRedundancy(...)` does not drive `Server.ServiceLevel` by itself, so Kubernetes readiness must also register a ServiceLevel provider, commonly with `AddServerServiceLevel(...)` or the leader-aware provider used by the sample.

## Redundancy shapes on Kubernetes

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also document the setup on Kubernetes that does not require any K8s integration (due to raft/crdt use)

Comment thread Docs/MigrationGuide.md
edits via a code-fixer.
- Open an issue on
[OPCFoundation/UA-.NETStandard](https://github.com/OPCFoundation/UA-.NETStandard/issues).

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove, new and not breaking change

Comment thread Docs/HighAvailability.md
Comment on lines +195 to +200
Documented limitations:

- The synchronous core `ISubscriptionStore` definition-persistence contract requires a synchronously-completing backend such as the in-memory or CRDT store. Full async persistence to a backend such as Redis would require an async `ISubscriptionStore`.
- `SharedKeyValueSubscriptionStore` restores definitions and retransmission state, but monitored-item data/event queues are not restored by `RestoreDataChangeMonitoredItemQueue` or `RestoreEventMonitoredItemQueue`.
- Continuation-point mirroring is best-effort. Built-in node-manager `ContinuationPoint.Data` is opaque and is not reconstructed on a backup; after failover a client may receive `BadContinuationPointInvalid` and re-issue Browse or HistoryRead, which OPC 10000-4 §6.6.2.2 permits. Node managers that can serialize their own continuation-point data may opt in through `IContinuationPointStore`.
- Deterministic EventIds are optional and only as stable as the event fields used. Alarms & Conditions clients should still call `ConditionRefresh` after failover as required by OPC UA.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Plan addressing the limitations

Comment thread Docs/HighAvailability.md
ct);
```

OPC UA does not standardize how active and backup clients exchange `SessionId` or subscription ids. The coordinator uses `ServerDiagnostics` when authorized; deployments that disable diagnostics should provide their own coordination channel.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets not leave this vague, this should be using the Raft/Crdt kv registered, not server diagnostics as it requires admin rights.

Comment thread Docs/HighAvailability.md

`UseRedundancyConsistency` registers a shared `IRaftConsensus` plus a native `RaftLeaderElection` (`ILeaderElection`). Raft leadership is decided by the consensus protocol itself — a single leader per term, no split-brain — which is stronger than the lease-CAS `SharedStoreLeaseElection`. Because the distributed features register their store and election with `TryAddSingleton`, call `UseRedundancyConsistency` first and they compose over the chosen store and election.

The consensus engine is pluggable through the `IRaftConsensus` seam. By default the DI registration uses a single-node [`RaftCs`](https://github.com/marcschier/raft-cs) replica (`RaftCsConsensus.CreateSingleNode()`, a real RaftNode with in-memory storage/transport that elects itself leader); `InProcessRaftConsensus` is a lighter deterministic in-process alternative for tests and in-process replica sets. For a multi-pod cluster, build a multi-node replica with `RaftCsConsensus.CreateCluster(nodeId, memberIds, transport, …)` — a `RaftCs.Transport.NanoMsg` transport (peers addressed by DNS) plus an in-memory or durable `RaftCs.Storage.File` WAL — and register it through `RaftConsensusFactory`. On Kubernetes, `UseKubernetesRaftConsensus` (in `Opc.Ua.Redundancy.K8s`) does this for you from the StatefulSet ordinal and headless-Service DNS. RaftCs ships its own `IRaftTransport` (NanoMsg), so Raft shares the NanoMsg substrate with the CRDT gossip layer. The `Applications/RedundantServer` sample (`HA_CONSISTENCY=strong`, `docker-compose.raft.yml`) is a runnable multi-node example.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove [RaftCs](https://github.com/marcschier/raft-cs)

…rSet

Adds an opt-in, gated server-side load-direction layer that answers a GetEndpoints request on a dedicated balancing discovery URL with the best peer's endpoints, complementing (never replacing) the standard client-driven RedundantServerArray/ServiceLevel selection.

- Core seam: IGetEndpointsDirector + StandardServer.GetEndpointsDirector property + a guarded hook in GetEndpointsAsync + hosted-service wiring (additive; default null = unchanged behavior).
- Opc.Ua.Redundancy.Server/Redundancy/LoadDirection: health ServiceLevel + separate load-weight gossip (publish/view with staleness age-out, fail-to-self, integrity/malformed drop, coalesced load), integrity-protected peer endpoint directory, BandedServerDirectionPolicy (health-tier -> least-load -> random), gated ServerLoadDirector (publish-own-on-normal-URL, redirect-on-balancing-URL, fail-safe), LoadDirectionStartupTask, UseServerLoadDirection DI, thread-safe tie-break.
- ServiceLevel stays a pure health/eligibility signal; a separate load weight drives A/A balancing. Consistency follows UseRedundancyConsistency; the high-churn load weight is always coalesced.
- Records are integrity-protected via IRecordProtector (verify-before-use, fail-closed).
- Docs: HighAvailability.md 'GetEndpoints load direction' section (usage, conformance caveats, consistency, security/threat model, pitfalls).
- 24 tests (view/endpoints/policy/director). Builds clean net10 + net48 (0 warnings); full Redundancy.Server.Tests suite 275 pass.
…/R2/R4)

- R2: LoadDirectionOptions.StrongEligibility routes the eligibility keyspaces (svc/ + endpoint/) to the linearizable Raft store in hybrid mode via a composable IStrongKeyspaceProvider that UseRedundancyConsistency aggregates; the high-churn load/ weight stays eventual. Hybrid store exposes IsStrongKey + DefaultStrongKeyPrefixes. + 2 routing tests.
- R1: LoadDirectionServerIntegrationTests drives a real StandardServer via ServerFixture with a configured ServerLoadDirector and a seeded peer; asserts GetEndpoints on the balancing URL redirects to the peer and a normal URL serves local. Validates the StandardServer.GetEndpointsAsync seam end-to-end. + 2 tests.
- R4: Applications/RedundantServer wires UseServerLoadDirection from HA_BALANCING_URL (StrongEligibility when HA_CONSISTENCY=strong); adds docker-compose.loaddirection.yml + README docs.
Full Redundancy.Server.Tests suite 279 pass; builds clean net10 + net48 (0 warnings).
Convert the subscription-definition lifecycle methods on ISubscriptionStore
to async so definitions can be persisted to an async network backend without
a sync-over-async wrapper:

- StoreSubscriptions -> ValueTask<bool> StoreSubscriptionsAsync(.., CancellationToken)
- RestoreSubscriptions -> ValueTask<RestoreSubscriptionResult> RestoreSubscriptionsAsync(CancellationToken)
- OnSubscriptionRestoreComplete -> ValueTask OnSubscriptionRestoreCompleteAsync(.., CancellationToken)

The per-monitored-item queue-restore hooks (RestoreDataChangeMonitoredItemQueue,
RestoreEventMonitoredItemQueue) stay synchronous because they run on the
synchronous monitored-item creation path.

SubscriptionManager awaits the new methods. SharedKeyValueSubscriptionStore now
awaits its shared-store writes directly (removes the fire-and-forget Complete/
Observe helpers). The Quickstarts DurableSubscription sample and the store/AOT
tests are updated to the async signatures. Docs (HighAvailability, migration
guide, remaining-work plan) updated.

Validated: net10 + net48 clean (0 warnings on changed libs); 279 Redundancy.Server
+ 145 Server subscription + 23 store tests pass.
Extend Applications/RedundantServer with a worked transparent-redundancy
deployment: two REDUNDANCY_MODE=transparent replicas that present ONE logical
OPC UA server behind ONE virtual endpoint.

- Program.cs: add HA_APPLICATION_URI (shared ApplicationUri - CreateSession
  validates the client serverUri against it), HA_SUBJECT_NAME + HA_PKI_ROOT
  (shared ApplicationInstanceCertificate) so all replicas present one identity.
- docker-compose.transparent.yml + nginx.transparent.conf: two replicas sharing
  one URI + one certificate (seeded into a shared PKI volume by server-a, reused
  by server-b) mirroring session + address-space state by CRDT gossip, behind an
  nginx TCP load balancer publishing the single virtual endpoint. Each replica
  advertises the load-balancer host (HA_HOST=lb); because that host is a DNS
  name the listener binds to all interfaces while discovery/CreateSession echo
  the one endpoint the client uses (ServerBase.FilterByEndpointUrl match).
- Docs: RedundantServer/README.md transparent section + settings; Docs/
  HighAvailability.md worked-sample pointer.

Compose validated with 'docker compose config'; sample builds clean.
Add INodeStateStore.EnumerateValuesAsync so a standby hydrates its local
graph in two streamed passes - EnumerateAsync for node topology, then
EnumerateValuesAsync for the latest variable values - instead of one
TryReadValueAsync round trip per variable.

AddressSpaceSynchronizer.SeedOrHydrateAsync now applies values from the
single streamed value pass, so hydrating a large address space costs a
bounded number of round trips against a networked (CRDT/Raft) backend
rather than O(variables). InMemoryNodeStateStore implements it by scanning
the value keyspace (same pattern as EnumerateAsync over the node keyspace).

Node topology is still materialized eagerly; snapshot/lazy topology
materialization for very large graphs remains a documented future
optimization (the CRDT path already exchanges a snapshot + deltas).

3 store tests added. net10 + net48 + AOT build clean; 282 Redundancy.Server
tests pass.
* http://opcfoundation.org/License/MIT/1.00/
* ======================================================================*/

#nullable enable

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove all #nullable enable from files in projects that are already <nullable>enable in their .csproj file

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rename back to .Kubernetes instead of .K8s (folder and project name and namespaces)

using System.Runtime.CompilerServices;

[assembly: CLSCompliant(false)]
[assembly: InternalsVisibleTo("Opc.Ua.Aot.Tests")]

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use the .csproj method <InternalsVisibleTo> like in all other projects (change everywhere assembly attributes are used for it)

/// to every node manager that opts in via <see cref="ILocalAddressSpaceSource"/>,
/// enabling active/active (multi-writer) replication of its address space.
/// </summary>
public sealed class CrdtAddressSpaceStartupTask : IServerStartupTask, IAsyncDisposable

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where possible change "Crdt" prefix of classes to "Replicated" across the PR. If there are conflicts use "CrdtReplicated" as prefix. It works nicer with the options name

throw new ArgumentNullException(nameof(eventState));
}

var builder = new StringBuilder();

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe better to binary serialize the nodestate and compute the Hash? Also, maybe use a faster hashing algorithm (e.g. not crypto)? Do some benchmarks.

Comment on lines +30 to +48
// ===========================================================================
// Adapter that binds the external Raft engine `RaftCs`
// (https://github.com/marcschier/raft-cs, shipped alongside the Crdt 1.1.0
// libraries) to the in-repo IRaftConsensus seam. Opc.Ua.Redundancy references
// the `RaftCs` and `RaftCs.Transport` packages and defines OPCUA_RAFTCS, so
// this type is compiled into the assembly. The guard remains so the file can
// be excluded if the RaftCs dependency is ever removed.
//
// Usage:
// - Single-node / in-process: RaftCsConsensus.CreateSingleNode().
// - Multi-pod: construct a RaftNode with durable storage
// (RaftCs.Storage.File) and a networked transport
// (RaftCs.Transport.NanoMsg, sharing the NanoMsg substrate with the CRDT
// gossip layer), then `new RaftCsConsensus(node)`. Wire either through
// RedundancyConsistencyOptions.RaftConsensusFactory (server) or the client
// AddRaftClientSharedStore/AddRedundantClientSharedStore factories.
// ===========================================================================

#if OPCUA_RAFTCS

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove including guard ifdef. Rename class to "DefaultRaftConsensus"

Phase 1-2 of snapshot+delta address-space hydration (fast time-to-ready).

- NodeStateChange gains a monotonic Sequence.
- InMemoryNodeStateStore embeds a single-writer monotonic sequence inside each
  protected n/ and v/ record; every read path parses it and CurrentSequence /
  ObserveSequence expose the high-water mark (reseed-on-promotion).
- New INodeStateSnapshotStore capability (optional; store falls back to the
  streamed path when absent): WriteSnapshotAsync builds a chunked, atomically
  published snapshot (snap/ + snapmeta/manifest, 1 MB chunks, predecessor-gen
  GC) and trims the bounded delta log; TryReadSnapshotAsync streams it back;
  every node/value/delete write appends dlog/<seq>, and ReadDeltaLogAsync
  replays entries after a sequence in order.
- InMemoryNodeStateStore is now IDisposable (snapshot semaphore); the registry
  already disposes registered stores.

net10 + net48 clean (0 warnings); 287 Redundancy.Server tests pass (5 new store
tests: snapshot round-trip, delta-log replay, trim, sequence high-water).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants